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Abstract 

We collect several observations that concern variable-length coding of 
two-sided infinite sequences in a probabilistic setting. Attention is paid 
to images and preimages of asymptotically mean stationary measures de- 
fined on subsets of these sequences. We point out sufficient conditions 
under which the variable length coding and its inverse preserve asymp- 
totic mean stationarity. Moreover, conditions for preservation of shift- 
invariant cr-fields and the finite-energy property are discussed, and the 
block entropies for stationary means of coded processes are related in 
some cases. Subsequently, we apply certain of these results to construct 
a stationary nonergodic process with a desired linguistic interpretation. 
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1 Introduction 



Let X and Y be a pair of countable sets, called here alphabets. Fixing A as 
the empty string, denote the set of nonempty strings over an alphabet X as 
x+ : = U„ eN X " and the set of a11 strings as X* := X+ U {A}. The set of one- 
sided infinite sequences x N — (xi)i^ — X1X2X3... is written X N and the set of 
two-sided x 1 = (xi) ie z — ...X-\Xq.XiX2— is denoted by X z . (Mind the bold-face 
dot between the 0-th and the first symbol.) Shorthands x n := (Xi)i<i< n and 
x l k := (xi)k<i<i denote substrings, whereas \x\ is the length of a string x. 

Subsequently, consider a function / : X — > Y* that maps single symbols 
into strings. We will extend it to /* : X* -> Y*, / N : X N -> Y N U Y*, and 
/ z : X z -> Y z U (Y* x ¥*) defined as 

/* (x n ) := f( Xl )f(x 2 )...f(x n ), (1) 
f(x z ) := f( Xl )f(x 2 )f(x 3 )..., (2) 
f{x z ) := ...f(x. 1 )f(x Q ).f(x 1 )f(x 2 )..., (3) 

where Xi G X. These extensions are known in literature under several names, 
such as "variable- length coding" or "sequence morphisms" jj. The finite 
extension JT]) plays a fundamental role in the definition of instantaneous codes 
in information theory 9]. On the other hand, probabilistic analyses that involve 
strong laws and ergodic theorems necessarily operate on infinite sequences, cf., 
e -g-j 24, ]jj 21 1 . For these analyses, extensions @ and ([3]) seem more natu- 



ral, and the variable- length coding) 3]) has been discussed by communication 
engineers for a few decades [7, 17 
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Fix a sufficiently rich probability space (51, J, P), and let (X^A*) and 
(Y z , y z ) denote the standard measurable spaces of two-sided infinite sequences. 
Let us consider a "shrunk" stochastic process [Xi)i^% : (0,^7) — > (X z , X z ) and 
an "expanded" process (Fi)igz : (£1, J) — t (Y z , y z ) related through an almost 
sure equality 

(Xi)i& = fdX^ez), (4) 

assuming that lim n |/*(X" m )| = lim m |/*(X™ m ) = 00 almost surely. Through- 
out the article, the distributions of these processes will be written as 

(i = P((Xi) i& e •) and is = P((Y t ) teZ e ■) = fi o (fy 1 . (5) 

Having denoted the shift operation as T(x z ) :— ...XQX1.X2X3... — (xi+i)j e z, 
a measure /i on (X z , X E ) is called asymptotically mean stationary (AMS) if the 
limits 



n — 1 

(A) = lira -Vjior^A) (6) 

n— >oo fi — J 



=0 



exist for all A 6 X , see [17|,|2l|. The limit £l, if it exists as as a total function 
X z — > K, forms a stationary measure on (X z , X z ), i.e., /2oT _1 = /2, and is called 
the stationary mean of [i. It is possible that limits ([6]) exist for a computable 
measure fi and all cylinder sets but they do not exist for some other sets, see 
Example 16.31 later. 
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As we shall show, under mild conditions, the transported measure v = /* o 
(/ z ) is AMS if jU is AMS. A weaker proposition, assuming a stationary fi, was 

obtained in [171 Example 6]. Besides coding theory, stationary means /i o (/ z ) 1 
of variable-length coded measures appear in disguise in statistical applications 
such as length-biased sampling II 01 or philosophical probabilistic puzzles such 
as the Sleeping Beauty problem [14j . 

An application at the interface between information theory and linguistics 
has drawn our attention to the question whether a few specific properties of 
a stationary measure /i can be simultaneously preserved by the stationary mean 
/jo (/ z ) 1 for a certain injection / z : X — > Y*, where X is infinite, and Y is 
finite. In this article, we gather several results of independent interest that 
concern partly relaxed and partly more general cases of our original problem. 
The question that stimulated our research will be presented at the end of this 
section and answered in positive later. 

We shall not discuss measures on one-sided sequences, see [22L [13], since 
they do not arise naturally in the methods and applications considered here. 
However, there appear a few more specific conditions on the coding function 
J z , which appeal to two-sidedness of coded sequences. The first condition has 
to do with various concepts of synchronization, see [26|, @, 0] • 

Definition 1.1 A function ir : X z — > Y z is called a synchronizable injection if 
7r is an injection and T t ir(x 1 ') = 7r(6 z ) for an i G Z implies T^x z = b z for some 

j'ez. 

For example, / z is a synchronizable injection for a comma-separated code 
f(x) = g{x)c, where c G Y m , and g : X -> (Y m \ {c})* is an injection. 

Other conditions considered are more local. Let us recall that a set of strings 
C C Y* is called (i) prefix-free if w =^ zs for w, z € C and s G Y + , (ii) suffix-free 
if w 7^ sz for w, z G C and s G Y + , (iii) fix- free if it is both prefix-free and 
suffix-free, and (iv) complete if it satisfies the Kraft equality J2 w ec |Y| = 1, 
where |Y| is the cardinality of Y. 

Definition 1.2 A function f : X — > Y* is called (complete) prefix/suffix/fix- 
free if f is an injection and the image /(X) is respectively (complete) 
prefix/suffix/fix-free. For finite f (X) , / is called finite. 

For instance, the set {01,000,100410,111,0010,0011,1010,1011} is complete 
fix-free with respect to Y = {0, 1} [16lll|. The aforementioned comma-separated 
code f(x) = g{x)c is prefix- free but it is not complete. 
The main results of this paper are as follows: 

(i) The measure /i o (/_ z ) is AMS for an AMS measure /i, provided that 
the expansion rate Z(x z ) := lim„ n~ x y^Lj is in the range (0, 00) 
/x-almost everywhere (Section [2]). This result generalizes fvA Example 6], 
where /1 o (/ z ) was shown AMS provided that fi is stationary, / z is an 
injection, and the /^-expectation of I is finite. 

(ii) The shift-invariant algebras for processes (-X"j)j £ z and f z ((Xi)i € z) remain 
in one-to-one correspondence, and their distributions coincide on these 
algebras if / z is a synchronizable injection (Section [3]) . 
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(iii) The measure v o / z is stationary or AMS respectively for a stationary or 
an AMS measure v if / is complete fix- free (Section 2]) . 

(iv) Write the cylinder sets as [u] :— {x z : a;' u = u}. As defined in [25j], a mea- 
sure p, has finite energy if conditional probabilities of cylinder sets are 
uniformly exponentially damped, i.e., if 

M(M) < #<>!/*([«]) (7) 

for certain c < 1 and K < oc. (Condition (0) may be only satisfied for 
c > |X|~ and, for a finite alphabet X, ([7]) implies that the length of 
the longest nonoverlapping repeat in the /i-distributed block of length n is 



almost surely bounded by O(logn) (25|.) We will show that the stationary 



mean ji has also finite energy if ([7]) holds. Moreover, /i o (/ z ) 1 has finite 
energy if \x has finite energy and / is finite prefix- free (Section [5]). 

(v) Block entropy for a measure [i on (X z , X z ) is the function 

H^i;n) := - £ A*(T _< [«D logM^M), (8) 

where we also use the shorthand H^(n) := 11^(0; n). We will demonstrate 
that for a fixed length injection / : X — > Y K , a finite X, and v = fio , 
block entropies (n) and Hp (nK) of the stationary means do not differ 
more than a constant (Section [6]). 

We have researched these topics while seeking for a class of nonergodic pro- 
cesses (Yi)iez that satisfy four conditions: 

(a) (yi)j 6 z is a process over a finite alphabet Y = {0, 1, D — 1}, 

(b) (Yi)i e z is stationary, 

(c) (li)iez has finite energy, and 

(d) there exist independent equidistributcd binary random variables (Zk)keN> 
P(Zk = z) = 1/2, z £ {0, 1}, measurable against the shift-invariant cr-field 
of {Yi)i £ z such that 

liminfn - ^ \Us(n)\ > (9) 



holds for a certain (3 £ (0,1), all 8 £ (1/2,1), and the sets Ug(ri) := 
{k £ N : P (sk (Y n ) = Z k ) > 8) of well-predictable Z^s, where functions 
Sk satisfy 

hm P (s k (Y^r) = Zk) = 1, V 4 e Z. (10) 

As demonstrated in [13] , properties (a)-(d) imply a power-law growth of the 
number of distinct nonterminal symbols in the shortest grammar-based com- 
pression of the block Y?£™, see 20, I]- In a linguistic interpretation posited in 



manuscript [l2j, variables Yl stand for consecutive letters of an infinitely long 
text, whereas the values of variables Zi stand for random facts repetitively de- 
scribed in the text. Since nonterminal symbols of grammar-based compressed 
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texts in natural language often correspond to words of a particular language 
(we mean words in the common sense of strings of letters separated by spaces), 
the demonstrated implication forms a new explanation of a power-law growth 



of text vocabulary (known as Zipf's law in linguistics 28]). Precisely, the ex- 
planation takes the form of the statement: If an n-letter long text describes inr 
independent facts in a repetitive way, then the text contains at least n^/logn 
different words. 

Properties (b)-(d), but not (a), are satisfied by the following process. 

Example 1.3 (0) Let (X t ) ie z be a process on (fi,j7, P), where variables 

Xi = {Ki,Z Ki ) (11) 

assume values from an infinite alphabet X = Nx {0, 1}, variables Ki and Z k 
are probabilistically independent, Ki are distributed according to a power law, 
P{Ki — k) = k~ a I 'C(ct) , a > 1, C( a ) '■— SfcLi k~ a , and Z k are equidistributed, 
P{Z k =z) = 1/2, zg{0,1}. 

Let us write u C v when a sequence or a string v contains a string u as 
a substring. For X = N x {0, 1} and v G X z U X*, define the predictors 

{0 if (k, 0)C» and (k, 1) % v, 
1 if (k, 1) C v and (k, 0) % v, 
2 else. 

Variables Zk are measurable against the shift-invariant cr-field of (Xj)j e z since 
they satisfy Zk = s k ((Xi)i e %) almost surely. Moreover, 

Lim P (s k (X%?) =Z k ) = l, ie Z, (12) 

and 



\U s (n)\ > 



l/a 



(13) 



-C(a)log(l-<5). 

for 5 e (1/2, 1) and U s (n) :={keN:P (s k (X n ) = Z k ) > 5}, as shown in [13]. 

We have supposed that a suitable distribution over a finite alphabet can be 
constructed as the stationary mean of a certain encoding of the process (fTTj). 
The results of Sections [2] through [5] suggest the following statement: 

Proposition 1.4 Let /j, = P((Xi)i^i G ■) be the distribution of the process from 
Example \1.3\ and put Y = {0, 1,2}. Consider the coding function / :Xh> Y + 
given as 

f(k,z) = b(k)z2, (14) 

where lb(k) G {0, 1} + is the binary representation of a natural number k. The 
process (Yi)i<z% distributed according to the stationary mean P({Yi)i e j d G •) = 

A* ° (/ Z ) 1 satisfies conditions (a) -(d) for (3 = aT 1 and C(a) > 4. Variables Z k 
may be constructed as Z k = s k {(Yi)i^£), where 

{0 if 26(/c)02 C w and 26(fc)12 % w, 

1 if 2b(k) 12 C w and 26(fc)02 % w, (15) 

2 else 

for wG Y Z UY*. 
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This proposition is proved in the final Section [7] The inequality C(/^ _1 ) > 4 
holds for (3 > 0.7728.... Mind that processes (Yi)i& an d (^Q)iez live on different 
probability spaces, say (fi,j7, P) and {Vl : J, P), respectively. Obviously, the 
coding function p4j) is prefix-free, and its extension / z is a synchronizable 
injection. 



2 AMS measures and finite expansion rate 

Previous accounts of AMS measures on two-sided sequences can be found in 



[171 1 1 51 ] - Let us recall a few useful facts. First of all, for the shift T(x ) :— 
...X0X1.X2X3... = (x l+1 ) te z, an AMS measure fi on (X z ,^ z ) can be equivalently 
characterized as such that the almost sure ergodic theorem is satisfied, i.e., the 
limit lim„ n^ 1 X^Lo* 9 ^ exists /i-almost everywhere for every nonnegative 
measurable function g : (X z , X z ) — > (R, TZ) [l7l . Theorem 1]. Trivially, ft = fi 
for a stationary /1. However, the equality 

fi(A) = n(A), A G I x , (16) 

is also satisfied for the T-invariant algebra 2x := {A € X z : T -1 A = A} 
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the general AMS case. This follows directly from ©, see 17 1. Extending 
the concept of ergodicity, usually discussed for stationary measures, an AMS 
measure fi is called ergodic if fi(A) £ {0, 1} for all A G %%. 
The lemma below is mostly a well known fact: 



Lemma 2.1 (cf. [15, Theorem 0]) A measure fi on (X £ ,X £ ) is AMS if and 
only if there exists a stationary measure r on (X z , X z ) such that r > [i. In the 
latter case, we have r ^ ^ \x. 

Remark: The notation r 3> fi stands for measure dominance, i.e., t(A) = 
implies that n(A) = for all sets A in the domain of [i. 

The proof in [l5[ does not cover the inequality r 3> p. To justify it, let us 
observe that t»/j and t(A) = imply fx(T~ l A) = 0. Hence fi(A) — as well. 
Moreover, the proof in [l5| cannot be carried to the one-sided case since it applies 
invariant sets of form T l A. The same trick resurfaces in Proposition 12.31 
below and in Section [31 where synchronizable injections are considered. 

By the definition, fi(A) = lim„ J n _1 J2*i=o ^-{T i x z eA} d^,(x z '). Hence 
a useful frequency interpretation follows by the dominated convergence. 
Lemma 2.2 For an AMS measure (i on (X z , X z ), 



p(A) 



^ 71— X 

lim - V l {T i x z e A} 

i=0 



dn{x L ), A G X z . (17) 



Remark: In the ergodic case, the integrated expression is almost everywhere 
constant. Symbol lfoy therein denotes the indicator function, i.e., '■= 1 if 
ip is true and := otherwise. 

Now we move on to variable-length coding of an AMS measure [i on (X z , X z ) . 
Let / : X -> Y*, ^(x z ) := |/(^)|, and S{x z ,n) := Yh=i Bv the ergodic 

theorem [l9|, Theorem 9.6] and Lemma I2~T1 the limit 

Itf) ~ lim _______ = E ^ (h || X x ) (x z ) (18) 
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exists both fi- and /i-almost everywhere. We will call the function l(-) the 
expansion rate, whereas its expectation will be denoted by 



L := I Idfi = / ld/.i = / lid/i. 



(19) 



Let T : (#*)»ez ^ (y»+i)iez a l so denote the shift Y z Y z . Put 

F(A, k, x z ) := l{T k / z (z z )eA}) 
1/0*01-1 

G(A,a; z ):= £ F(A,M Z ), 

where 53fc=o : ~ ^ - ^ ^ ne quasiperiodic identity 

T\f^f(x z ) = f z (Tx z ), 

we have 

F(T _1 4 fc, x z ) = F(A, k + l,x z ), 
F(A,\f( Xl )\,x z ) = F(A,0,Tx z ), 



G(T~ 1 A, x z ) = G(A, x z ) - F(A, 0, ; 



F{A,Q,Tx z ). 



(20) 
(21) 
(22) 



The following proposition is a direct consequence of the above identities and 
the two previous lemmas. 

Proposition 2.3 Let \i be an AMS measure on (X z , X z ) and suppose that the 
expansion rate U8\) is [i-almost surely in the range (0,oo) for an f : X — > Y*. 
Then v — - /io (/ z ) 1 is an AMS measure on (Y z ,y z ) with the stationary mean 



u(A) 



lim — \ 

PC**)]" 1 

[my 1 



S(x z ,n)-1 

F ^ k - 



k=0 
n-1 



lim iVG(A,T J x z ) 

n— »-co 77. * — ' 
i=0 

^ n — 1 

lim -VG(i,fi z ) 

n— »oo 7i ^ — ' 



i=0 



^(a; z ) 
d/i(x z ) 
d/2(x Z ), ie/ 



(23) 
(24) 
(25) 



Proof: The transported measure v is a measure on (Y z ,3^ z ) if and only if 
lim ra 5(x z , n) — - oo ^-almost surely. This condition is satisfied. Observe that the 
limits in the brackets in (|2"3")) - (r2"5"l) exist fi- and /2-almost surely. Consequently, 
the integrals are equal. Denote the right-hand side of ([2"3"]) as t(A). The function 
r is a stationary measure on (Y z ,3^ z ) by the dominated convergence and the 
Vitali-Hahn-Saks theorem. Suppose that there exists a set A £ 3^ Z such that 
v{A) > t(A) = 0. Then we would have u(B) > t(B) = for B = \J ieZ T z A. 
But B is shift-invariant, so t(B) = v{B) by formula (f2"3")) . Thus, our assumption 
was false, and we rather have t v. Hence v is AMS in view of Lemma \2. II 
Moreover, r coincides with the expression for v given by Lemma 12.21 □ 
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Corollary 2.4 If the expansion rate H8\) is p-almost surely in the range (0, oo) 
and p = f for two AMS measures p and r, then 



Putting r = p, we obtain v — fi o (/ z ) _1 and v 3> po (/ z ) 1 for v = /io (/ z ) . 

Under much stronger assumptions, there exists a "finite-sum" expression for 
the stationary mean D in terms of p, noticed by Kieffer and Gray for p, = p 



17, Example 6]. Namely, we can construct a stationary measure p by averaging 



the stationary mean p o (/ z ) over a randomized shift within the quasiperiod 
|/(sgi)|. This idea is more generic, see [7L fTsjl . 

Proposition 2.5 Let p be an AMS measure on (X z , X z ) and suppose that the 
expected expansion rate U9\) is in the range (0,oo) for an f : X — >• Y*. Then 
there exists a stationary measure 

p{A) =jf G(A,x z )dp(x z ), A G y z . (26) 

Remark: Under the above assumptions, the expansion rate l(-) may vanish on 
a set of positive measure. 

Proof: Stationarity of p was discussed in pjj for an injective / z . The following 
proof is more general. First of all, we have p(Y z ) = 1, whereas the countable 
additivity follows by the dominated convergence theorem. As for stationarity, 
we obtain 

p{T- l A) - p(A) = L- 1 j (F(A, 0, Tx z ) - F(A, 0, x z )) dp(x z ) = 

from (EH) and p o T" 1 = p. D 

Although p does not necessarily equal v, it dominates the measure v. 

Corollary 2.6 Suppose that the hypothesis of Proposition \2.5\ holds true and 
v = p o (/ z ) is a measure on (Y Z ,3^ Z ). Then v is AMS and p 3> P,p,o 

Proof: Observe that p(A) > L _1 /2((/ z ) _1 A). Hence p > p, o (/ z )~\ Since 

p o (/ z ) '»p (/ z ) 1 = v follows from p^> p, we obtain p~^> v. Measure p 
is stationary by Proposition ^. 51 so v is AMS, and p 3> P by Lemma |2~T1 □ 

The next proposition states that p is the stationary mean of the transported 
measure if the expansion rate is almost surely constant. 



Proposition 2.7 Suppose that the hypothesis of Provosition \E75\ holds true and 
l(-) = L p-almost everywhere. Then p = v for v = p o (/ z ) 

Proof: By stationarity of p, identity (f2"2")l . and the dominated convergence, 
p(A) = L- 1 J [lim„ n" 1 ^0 G(A, T^ z )] dp(x z ). 

This expression equals v(A) by Proposition 12 . 31 if £(•) — L almost surely. □ 

Example 2.8 The equality /(•) = L holds almost everywhere for the nonergodic 
process fTTJj and / : N X {0,1} -> {0,1}* i/ /(fc,z) = g(k)w(z) for k G N, 
z G {0, 1}, |<?(fe)| = O(logfc), and = A. Code fi^|) /aZfe under that case. 
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3 Synchronization and shift-invariant cr-fields 

For an injection tt : X z — >• Y z , the transported shift 

T T := ttoTott" 1 , (27) 



considered in [IT], Example 6], constitutes an injection 7r(X ) — > 7r(X ). In that 
case, v :— fi o 7r _1 is stationary with respect to Tr for a stationary measure /i 
on (X z , X z ), i.e., voT, 1 = i/. 

We will demonstrate that ergodic properties of measures /i and v — ^ o 7r _1 
may be further related in the more specific case of a synchronizable injection. 
Some apparent technical difficulty is that the set 7r(X z ) usually does not belong 
to the T-invariant algebra Xy := {A G y 2 : T~ 1 A = A}. However, this can be 
overcome easily given certain care. 

Lemma 3.1 For an injection tt : X z — > Y z , consider pseudo-invariant algebras 

Q := {A G y z : A = B n tt(X z ), T^B = 5} , 
:= {A e y z : A = B (1 tt(X z ), Tr 1 ^ = B} , 

where T w is defined by We have 

QcQ, = vr(Ix). 

Proof: The right equality is obvious. As for the left relation, observe that 

Ui eZ Ti £ => \} l&L T l B D B - If T ~ lB = B then U e z T * B = B - Hence B n 
7r(X z ) G <2„. since formula {J i&z T*B defines a T„.-invariant set. □ 

Proposition 3.2 For a synchronizable injection tt : X z — > Y z , 

G=G*. 

Proof: By Lemma [3.11 Q C Q T . Thus it suffices to show that Q„ C Q or, 
equivalently, that X-% C 7r _1 (Q). We will demonstrate the latter. Consider 
anAglx and construct the set E = 7r(X z ) n lJ ieZ T i 7r(A) G Q. Since tt is 
synchronizable and A is T-invariant, we have that 7r _1 (T) = A. □ 

Proposition 3.3 Consider a synchronizable injection tt : X z — > Y z 7 a measure 
/i on (X z , A*), and its image v = ji o 7r _1 on (Y z , 3^ z ). For eac/i Xv G Xy, t/iere 
exists such an A G Xx, and /or eac/i A G Xx, i/iere exists an E G Xy smc/i £/ia£ 

^(T) = f(.E ("1 7r(X z )) = fi(A). (28) 

Remark: As a further corollary, either both measures fi and are ergodic or 
neither of them has this property. Let us recall that p(A) = fi(A) for A G Xx and 
an AMS fx. The analogical equality P(-E) = v(E) holds for B G Xy and an AMS 
i/. Some oddness of (|28[) is buried in the fact that v(E) does not necessarily 
equal v(E n 7r(X z )). It is only the support of v that is confined to 7r(X z ), and 
7r(X z ) need not be T-invariant, as it has been remarked. 

Proof: For E G X Y , take A = tt^ 1 (E). For A G X x , take £ = \J ieZ T l ir(A). 
Then the equality follows immediately from Proposition 13.21 □ 
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Example 3.4 Process ill]) has a nonatomic shift-invariant sub-a-field flS] . 
Hence the expanded process (li)iez = f z ((Xi)i £ z) distributed P((Yi)i e j J S •) = v 
and its stationary mean distributed P((5^)jgg 6 •) = 77 /icwe i/ie same 

property if we use a comma- separated code f : X — > Y* , like [Lfy . Moreover, 
(^)ieZ /mw a nonatomic shift-invariant sub-a-field if and only if 110\) holds true 
for certain functions Sk and independent equidistributed binary random variables 
(Z fe ) feeN J7j, Theorem 9]. 



4 Complete fix-free codes and stationarity 

This section contains a result of independent interest, loosely related to the 
setting of our initial problem. For any injection 7r : X z — > Y z , measures may 
also be transported in the opposite direction. That is, for any measure v on 
(Y z , y z ), there exists a measure [i := von on (X z , X z ). A condition opposite to 
synchronization appears when this mapping is required to preserve stationarity. 
We came across the following proposition, which seemingly has not been noticed 
so far, cf. 0| : 

Proposition 4.1 Suppose that X is finite and f : X — > Y* is complete fix-free. 
Then 

(i) / z is a bisection X z -> Y z , and 

(ii) fi = vo / z is a stationary measure on (X z , <Y Z ) if v is a stationary measure 
on(Y z ,y Z ). 

Remark: Statement (i) may be false for a complete infinite prefix- free set /(X). 

For instance, the set ({...00.00..}) is empty for f(n) = {0™- x l : n e N}, 

X = N, and Y = {0, 1}. However, we do not know of any complete infinite set 
of strings that would be both prefix- and suffix- free, cf. [lil H|. 

Proof: Let £ = /(X). 

(i) Clearly, \w\ > 1 for w £ £ if / is a bijection. Thus / z (a; z ) is a two- 
sided sequence for x z € X z . Moreover, given a y z £ Y z , we can reconstruct 
the unique x 1 £ X z with f z (x z ) = y z by cutting off the consecutive suffixes 
or prefixes belonging to L from and By the following reasoning, this 
parsing process is guaranteed not to stop after a finite number of steps. 

On the contrary, assume that there is such an infinite sequence (the 
mirrorlike argumentation applies to y^^) such that no w £ C is a prefix of 
Let v be a prefix of that is longer than any w £ C. The set {v} U C is 

prefix-free, and so J2wec W — — 1^1 < 1 by the Kraft inequality. We 
have arrived at a contradiction, so the assumption was false. 

(ii) By the Kolmogorov process theorem and the tt-X theorem, stationarity 
of /i is equivalent to the set of equalities 

X>([H)= KM) = (29) 
zee zee 

On the other hand, stationarity of v is equivalent to 

^KM)=KM)=£KM), weY*. (30) 

seY s£Y 
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The following auxiliary fact is useful to derive from ([50]) : Let 1{M) > 1 
be the length of the longest string in a set M.. Any finite complete prefix- free 
set M C Y* may be decomposed as M = M r U (M p x Y), where l(M) = 
l(M p x Y) > l(M r ), and A4 m = 7W r U M p is a complete prefix-free set. This 
decomposition may be proved by contradiction with the Kraft inequality applied 
to M m . 

Fron this and from (f3T)| , it follows that for any complete prefix- free M. with 
l(Ai) > 1, there exists a complete prefix-free A4 m such that l(M) — l(Ai m ) — 1 
and 

v i\ wz \) = "(H)- 

zeM z£M m 

Using this, the left equality in (|29p may be proved by induction on l(M.) starting 
with Ai m — {A}. The proof of the right equality is mirrorlike. □ 

Corollary 4.2 Suppose that X is finite, f : X — > Y* is complete fix-free, and v 
is AMS on (Y z , y z ). Then fj, = v o f z is AMS on (X z , X 1 ) with the stationary 
mean /i<i/o/ z . 

Proof: We have /i = vo/ z «i/o / z , where the last measure is stationary by 
Proposition 14. II Hence the claim follows by Lemma H2TT1 □ 

5 Preservation of the finite-energy property 

We supposed that both / z and (/ z ) 1 preserve the finite-energy property if the 
coding function / is sufficiently nice, prefix-free in particular. The proofs are 
a bit more complicated than we expected, but convenient sufficient conditions 
can be formulated. 

Definition 5.1 More specifically, we will say that (i) a measure fi on (X z , X z ) 
has (K, c)-energy if c < 1, K < oo, and condition ^ holds and (ii) the measure 
fi has (K, c, /)-energy for a coding function / :X— >Y* if c < 1, K < oo, and 

MM) < Kc^^fi([u\). (31) 

Remark: If a function / : X — > Y* is prefix-free, then, by the Kraft inequality 
J2 X £X \Y\~^^ < 1, condition (|3Tj) may be only satisfied for c > |Y| _1 . In 
particular, the inequality c > |Y 1 must be strict for a noncomplete coding 
function, i.e., when X^ex lYp''^''' < 1. 

Proposition 5.2 If f : X — ► Y* is prefix-free and a measure [i on (X z , X 1 ) has 
(K,c, f)-energy, then /i has also (K,c)-energy. 

Proof: If / is prefix-free, then \ f*(u)\ > \u\. Hence, 

H([vu]) < Kc lr{u)l n([v}) < Kc^n([v]). 

□ 

Proposition 5.3 If f : X — > Y* is prefix-free and a measure v — [i o (/ z ) 1 on 
(Y z , y z ) has (K, c)-energy, then the measure [i on (X z , X z ) has (K, c, f)-energy. 
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Proof: Let i/([zw]) < Kc^ w)[ v(\z\). The extension /* is an injection for a prefix- 
free /. Moreover, if both z and w belong to /*(X*), then 

{fy 1 ([zw]) = [(rr l ( Z ) err 1 hi- 

In particular, for z = f*{u) and w = f*(v), we obtain 

/x(H) = KM) < Kc M v([z}) = if C l / *WI /i ([ u ]). 

□ 

The converse of Proposition l5T51 is valid under additional restrictions. Denote 
the difference of strings w, z G Y* as 

!s if w — zs, s G Y + , 
A if z = ius, s e Y*, 
w else. 

We choose this definition to have pi™ -2 < pM~M for p G (0, 1). The set of z's 
that fall under the first two cases is denoted as 

(w) := {z G Y* : 3 s£ y« : w — zs V z = ws} . 

Moreover, for a set C G Y*, we define the remainder u>£ of a string w G Y* as 
the shortest element of the set {w — s : s G £*, |s < The completion set 

of the string w with respect to the set C is defined as 

C w := {s G Y* : w c s G £} . 

This set is prefix-free for a prefix-free C. 

Proposition 5.4 Suppose that f : X — > Y* is prefix-free and a measure fi on 
(X z , has (if, c, f)-energy. For C = /(X), put 



M f (p) := sup ^ p 



If Mf(c) < oo and Nf^icz) < oo /or a certain c 2 G [c, 1), i/ien the measure 
v = M° on (Y z ,y Z ) has (K , c 2 )- energy, where K = N f ^(c 2 )M f (c)K. 

Proof: If z = f*(u) and tu = /*(u) for some u and u, then 

= M(H) < Kc ]r{v)] (j,([u]) = Kc^u{[z\). 

Notice that v([z\) = J^sec ^d 2,3 ]) f° r an y z e Assume now that z G L* 
and let to be arbitrary. By C zw = £ lu we obtain 

v([zw\) = J2 "(l>H) < 1] ifd wa li/([«]) < Af / (c)^cl"'l i y([z]). 
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Eventually, consider arbitrary z and w. We have 

v{[zw])= v{[zs{w~s)]) < ]T M f (c)Kc^v([z S }) 

s£C z n{w) sec z n{w) 

< Y M f (c)Kc l 2 w - s ^([zs}) < M f (c)Kc[ wl £ c^'KN) 

>l, 



□ 



Corollary 5.5 // / : X — > Y* is finite prefix- free and a measure fj, on (X z , 
ftas finite energy, then the measure v — fio (/ z ) 1 on (Y z , y z ) has finite energy. 

Proof: Assume that fj, has (K , q)-energy. We have \C W \ < \C\ < oo and 
su Pze£„ M — su Pzec M < 00 • Hence we obtain inequalities Mf(p) < oo, 
Nf^(p) < oo, and (pITj) for p < 1 and c € [max zg £ g 1 /' 2 ', 1). In consequence, the 
claim follows by Proposition 15.41 □ 

Below we present a more specific example with an infinite image /(X). 

Corollary 5.6 Let / : X = N n- Y+ = {0, 1, 2} + be given as 

f(k) = b(k)w(k)2, 

where lb(k) G {0,1} + is the binary representation of a natural number k, and 
w(k) G {0,1}* is a string of fixed length, \w(k)\ = A. Let also fi be a measure 
on (X z , X 1 ) that satisfies fj,([vu]) = and 

/*([*]) = ^4, ken, 

for some a > 1. If C( a ) > 2 A+1 , then /j has (l,c, /) -energy for c E 
[max{2- Q ,(C(a))" 1/(j4+1) } ,2-!), w/iereas v = [io (/ z ) _1 /ias (K ,c 2 )-energy 
for K = N f ^(c 2 )M f (c) andc 2 E (max {c, 2^ Q } , 1). 

Proof: We have |/(fc)| = |log 2 fcj + 1 + |w(fc)| and fc~ Q = (2- Q ) lo fe fc . Thus, 

(C(a)r N (2- Q ) lr(tl)| - A|u| < jt([u]) < (C(a))-l n l(2- Q )l r (")l-( A+1 )l tl l < c lr(u)l . 

In particular, (f5T|) follows for K — 1. Consider a string to G Y* and let a; be 
the number of strings of length I in the set C w . We can see that a; < 1 for 
I < A + 1, whereas a; = 2'~(' 4+1 ) for Z > A + 1 if £ w is not empty. Hence, 

M f( c ) < J2Zo uiayi { 1 ^ 2l ^ {A+1) } cl < 00 and 

So the claim holds by Proposition [5T31 D 

By means of the following two simple statements, the above result can be 
extended to certain nonergodic measures, including the distribution of process 
(fTTj) and its stationary variable-length coding. 
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Proposition 5.7 Consider a measure P on (CI, J) and a probability kernel r 
from (CI, J) to (Y z ,y z ) (i.e., t(-,uj) is a measure on (Y z ,y z ) for P -almost all 
uj € CI, and the function r(A, ■) is measurable J for each A £ y 1 ). If t(-,u>) has 
(K, c)- energy for P- almost all uj € CI, then so does the measure J r(-,uj)dP(uj). 

Proof: 

/ T([zw],uj)dP(uj) < J Kc\ w W([w\,io)dP(uj) < KcW J T([w],w)dP(w). 

□ 

Proposition 5.8 If an A MS measure v on (Y z ,3^ z ) has (K,c)-energy, then so 
does the measure v. 

Proof: 

p,([zw]) = lim„ n- 1 Y^=o £ sG y* f*([ szw }) 

< KcM lim„ n" 1 Et"o E,^ MM) < Kc^fl([z}). 

□ 

6 Block entropies of stationary means 

For a stationary measure \x, block entropy H^(n) = H^(0;n) denned in ([5]) is 
a nonnegative, growing, and concave function of n, see Hence the limit 

h ■= r lm (32) 

71— >oo 77, 

known as the entropy rate, exists in that case. Whereas block entropy behaves 
less regularly in a general AMS case, we can bound the block entropy of the 
stationary mean in the following way. 

Proposition 6.1 For an AMS measure fi, 

^ n— 1 

Hp,{m) > limsup — PI^(i] rn). (33) 

n— >oo ri 

Proof: The claim 

n — 1 

-EE M(T->])log M (T->]) 

follows by the Jensen inequality for the function p i-> — plogp. □ 
Proposition 6.2 Let \i be AMS with hp < oo and H^ri) < oo /or a/Z n. Then 

h < Umill f^M. (34) 

n— >oo 77, 



E M(N) log m( [u] )> limsup 
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Proof: By the generalized Shannon-McMillan-Breiman theorem [B|, Theorem 
2] and (fTrJ)) . the L 1 ^) convergence 

h a = I lim jgjflET) ^) = _ / lim l ^& Mx Z) (35) 



holds for the stationary measure ll if < oo. On the other hand, by [B], 
Theorem 3], an AMS measure n with H^n) < oo satisfies 



logn([x n ]) log/j,([x n ] 
lim = lim 



for ll- and /i-almost all x z . Hence the claim follows by the Fatou lemma. □ 



(36) 



Example 6.3 Using Provosition \6.2l we can show a simple example of a mea- 
sure y, such that limits 0) exist for all cylinder sets but ii is not AMS. Consider 
the Champernowne sequence 6 N = 12345678910111213... (i.e., the concatenation 
of decimal representations of natural numbers) and put fi([b k ]) = 1 for k G N. 
We have p,([u]) = lO - '™' since 6 N is normal. If ft is extended from these val- 
ues to a measure on (X Z ,<Y Z ), then Hp,(m) — to log 10 and hp = log 10 but 
H^{i;m) = 0. Hence fi cannot be AMS. 

Block entropies of two stationary means linked through variable-length cod- 
ing can be related as well. The link for the entropy rate is very simple if the 
expansion rate is constant and the coding function is uniquely decodable. 

Proposition 6.4 (cf. (13, Theorem 1]) Let ll be AMS on (X z , X z ) with 
hp < oo and H^(n) < oo for all n and suppose that the expansion rate satisfies 
l(-) = L G (0, oo) ^-almost everywhere for a prefix-free f : X — > Y*. Then we 
have 



L-'h, 



(37) 



for the measure v = jj, o (/ z ) 1 if hp < oo. 

Remark: We have hp < oo and H^{n) < oo if ll is stationary and H^l) < oo, 
whereas hp < oo if the alphabet Y is finite. Formula ((37]) is a special case of 27J, 
Theorem 1] , but their proof is partly flawed. It uses the version of the Shannon- 
McMillan-Breiman theorem [l7|, Corollary 4] , which is true only for finite X and 
Y. A correct proof for any X and Y, invoking the already mentioned Theorems 2 
and 3 from is given below. As noticed in |27l . Theorem 1], lim n n~ x log Li([x n ]) 
converges almost surely to the entropy rate of the x z -typical ergodic component 
of the measure ll also when the expansion rate is not constant, see the ergodic 
decomposition theorems in [ljj Chapter 9 until Theorem 9.12]. 

Proof: The measure v = ll o (f z ) 1 is AMS by Proposition 12.31 whereas the 
extension /* is an injection for a prefix-free /. Hence /x([cc ra ]) = v([f*(x n )]) 
and \f*{x)\ > \x\. As a result, H v {n) < H^(n), whereas (j3"5j) . (|3"6"j) . and their 
analogues for v imply 



hp = - 



lim rrMogi/Qy"]) dv(y L ) 

n—>oo 



lim |/*(a:")r 1 log/x([a! n ]) d»(x z ) 
= - I \Ux I ')r 1 \ lim n - 1 ]ogu([x n ])\ <^(x z ) = L~ 1 hn. 
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□ 



In the following, we wish to obtain some bounds for H e (n) in terms of Hp (n) 
for finite n. We shall observe that formula can be interpreted in terms of 
random variables if \f(x)\ > for all i£X. To simplify the notation, we shall 
assume here that /2 = /i := P((X;)j e z G •) is the measure of a stationary process 
(Xi)i e z- Then p — P((Yi)i £ z G •) is the stationary distribution of 



(Y i ) ieZ = T"r((X i ) ie z), 



(38) 



where the random shift N and the nonstationary process (Xj)jgz are condition- 
ally independent given X\, their distribution being 



p{x{ = 4) = p{x{ = 4) 



P{N = n\Xi = x x ) 



L {0<"<l/(a:i)|-l} 



k<l<l, (39) 
n e N U {0} . (40) 



Suppose that l(-) = L holds /^-almost surely. As shown in Proposition ^. 71 this 
guarantees that p = v for the AMS measure v := P((Yi)i e % G ■) of the expanded 
process (jl]). Thus we have 

H 9 (n) = Hp(Y k k+n ~ 1 ) and H n {n) = ^(A^ 1 ), (41) 

where Hp(U) := Ep [— \ogP(U — •)] is the entropy of a discrete variable U. 

Denote the conditional entropy Hp(U\V) := Hp(U, V) — Hp(V) and covari- 
ance Covp(U,V) := Ep(UV) — Ep U~Ep V. Entropies of blocks drawn from the 
above introduced processes can be linked easily when blocks of random length 
are allowed. 

Proposition 6.5 Suppose that a process (Xj)i e z is stationary and L = 
Ep \f(Xi)\ < co for a prefix-free f : X — >• Y*. Consider then processes (li)ieZj 
(Xi) ie z, and {Yi) i£ z that satisfy WB, and [fify . Put also M n := 



Eti\f(^)l M n 
Then we have 



EJU /(**) , and V := E P 



l/(*i)l 

L 



log 



l/(*i)l 
L 



> 0. 



(i) Hp(Y M ») = H P (X n ), 



(ii) Hp(X l k ) > H P (X l k )- V if and only ifCavp(\f{X{)\ , -logP(X l k = •)) > 
for k < 1 < I, 

(Hi) Hp(N\X l k ) = logL + T), k < 1 < I, whereas 

(iv) H P (Y M "- Ml ,N) < H P (X n ,N) and Hp(X$,N) < Hp(Y^,N). 
Proof: In (i) and (iv), we use that /* is an injection for a prefix- free /. 

(i) The claim is true since Y M " = f*(X n ) and X n = (/*) _1 (F M "). 

(ii) Whenever Cov P (\f(Xi)\ , - logP(A£ = •)) > 0, we observe 



Hp{X l k ) 



V = e p 
> E P 



\f(Xi)\ 



^logP(X[ 



l/(*i)l 

L 



Ep [-\ogP{X{ = -)}=Hp{X k ). 
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(iii) By 601). 

(iv) By ([551) . the string y Mn ~ Ml is a function of I" and N, whereas string 
X% is a function of Y M ™ and N. Hence the claimed inequalities follow. 

□ 

Corollary 6.6 Let p be AMS on (X Z ,X Z ) and f : X Y L be a prefix-free 
fixed-length coding. Then we have 

\H B {nL) - H fi (n)\ < ff A (2) +logL (42) 
for the measure v = jj,o (/ z ) . 

Proof: In view of Corollary 1 2. 41 we may assume without loss of generality that 
fj, is stationary Let us repeat the construction of processes that precedes Propo- 
sition Observe that the processes X l k and X l k share the same distribution 
and M n = nL. Thus Proposition Hj^iv) yields Hp{Y^ l - lS > L ) < Hp_(X n ,N) < 
H P (X n - 1 ) + H P {X 2 _,N) and HpiX"- 1 ) = Hp(X^ 1 ) < Hp(Y Ln , N) < 
Hp(Y L ( n - 1 '>) + Hp(Y L _,N) < Hp{Y L{n -V) + Hp{X 2 ,N). To complete the 
proof, notice that Hp(X 2 , N) = H P (X 2 ) + logL. □ 

7 Encoding of the process X{ = [K^Zx^) 

We have not managed to produce an analogue of Corollarv l6.6l for the processes 
discussed in Proposition [TU But, as shown in [T^, then we have 

H P {X n ) > h^n + [log 2 - V (5)] ■ \U$(n)\ , (43) 
H P (Y m ) > h i> m+ [log 2- r](6)] ■ \U s (m)\ , (44) 

where v := jj, o (/ z ) 1 and rj{p) := — plogp — (1 — p)log(l — p). Whereas 
hp = L~ 1 h tli by Proposition ^. 41 point (d) of the proof below demonstrates that 
Us(m) D U s (n) for S > 5/a, n = [(6 - S/a)L^ 1 (m - C a )\ , a £ (6,1), and 
a certain constant C a . 

Proof of the Proposition ITT41 

(a)-(b) Process (\f(Xi)\)i £ z is ergodic. Thus the expansion rate equals its expec- 
tation almost surely: 

°° k~ a 
r((Xi) ieZ ) = L = Ep \f(Xi)\ = V(Llog 2 fcj + 2)— G (0, oo). 



Hence the stationary mean p o (/ z ) exists by Proposition 12.31 and con- 
stitutes a measure over a finite alphabet. 

(c) Consider the process (li)iez — f Z {{Xi)i£z) and the probability kernel 
t(-,cj) = P((Yi) ieZ e ■\\(Z k ) keN )(uj). For ((a) > 4 and P-almost all w, 
t(-,cj) takes form of the measure v considered in Corollary 15.61 Hence 
the distribution of the process (i^)igz has finite energy by Proposition ^. 71 
and, consequently, (Y)igz has finite energy by Proposition [^FJ 



16 



(d) Define Z k '■= s^((li)j e z). The functions s k are shift-invariant. Notice 
that Zk = Sk((Yi)i£z) almost surely on the space {Vt,J,P). Hence, by 
Eq. (|T6| applied to the AMS measure /j, o (/ z ) , the process (Zk)keN 
also consists of independent equidistributed binary variables measurable 
against the shift-invariant cr-ficld of (l^)j e g. 

Repeat the construction of processes that precedes Proposition 16 . 51 
putting M n := E?=i\f( x i)\ and M n := £™ =1 \f(X % )\. Recalling that 
Ep Mi = L, fix such a C a > that 

E P [Afil {Ml < Ca} ] > ah 

for some a e (<5, 1). Observe that s k (Y m ) = z if s fc (A7, 1 ) = z G {0, 1} and 
M„ < m. Hence, 

P (s k (Y m ) = Z k )>P (s k (X$) = Z k ,Nh < C a , M n - Mi < m - C a ) . 

The event on the right-hand side is measurable (Xj)jgz since Z k = 
Sk{{Xi) i& ). On the other hand, Sfe(pQ) ieZ ) = Z k . Thus by |3S]) and 
further by the independence of the variable Mi from (X^ 1 , Zfc) we obtain 

P (s k (X%) = Z k ,Mx < C a , M n -M x <m- C a ) 



= L _1 Ep 



-™ 1 ■'"{sfc (X%) = Z k ,Mi <C a ,M„ -Mi <m-C a } 



1 {s fc (X 2 ") = Z fc ,M„-Mi<m-C a } 



= L 1 Ep [Mil{Mi<C„}] Ep 

But is stationary, so the last expression yields simply 

P (s fe = Z k ) > aP (s k (X n ) = Z k , M n <m-C a ). 

Now, by P(A CiB)> P(A) - P(B C ) and by the Markov inequality, 

P(s k (X n ) = Z k ,M n <m~C a ) 

> P (s k (X n ) = Z k ) ~ P (M n >m-C a ) 

>P(s k (X n ) = Z k ) 



m - C a 

Taking 5 > 5/a and n = \{5 — 5/a)L~ 1 (m — C a )\ , we obtain 

k g U s {m) a(P (s k (X n ) = Z k )-(S- 5/a)) > 5 
P (s k (X n ) = Z k )>5 ke U s {n), 

so © follows for /3 = a -1 from 1(15)1 . 



□ 
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